> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/mlfoundations/open_clip/llms.txt
> Use this file to discover all available pages before exploring further.
# Training Configuration
> Complete reference for all OpenCLIP training parameters and hyperparameters
## Overview
OpenCLIP provides extensive configuration options for training CLIP models. This page documents all important training flags and hyperparameters from `params.py`.
To see all available options:
```bash theme={null}
python -m open_clip_train.main --help
```
## Data Configuration
### Training Data
Path to training data. For WebDataset, use glob patterns like `/data/train-{0000..2175}.tar`. Multiple sources can be combined with `::`.
```bash theme={null}
--train-data "/data/cc12m/train-{0000..2175}.tar"
--train-data "/data/cc12m/train.tar::/data/laion/train.tar" # Multiple sources
```
Path to validation data (same format as train-data).
```bash theme={null}
--val-data "/data/val.csv"
```
Total number of samples in training dataset. Required for WebDataset.
```bash theme={null}
--train-num-samples 10968539 # CC12M
```
Number of samples in validation dataset.
Dataset format: `webdataset`, `csv`, `synthetic`, or `auto` (auto-detect).
```bash theme={null}
--dataset-type webdataset
```
Enable sampling with replacement for webdataset. Recommended for large datasets and multiple data sources.
```bash theme={null}
--dataset-resampled
```
### CSV Data Parameters
Column separator for CSV files (tab by default).
```bash theme={null}
--csv-separator "," # Use comma separator
```
Column name for image paths in CSV.
```bash theme={null}
--csv-img-key filepath
```
Column name for captions in CSV.
```bash theme={null}
--csv-caption-key title
```
### Data Upsampling
Upsampling factors for multiple data sources, separated by `::`. Controls relative sampling probability.
```bash theme={null}
--train-data "/data/cc12m/train.tar::/data/cc3m/train.tar" \
--train-data-upsampling-factors "1::4" # Sample CC3M 4x more frequently
```
## Model Configuration
### Model Selection
Model architecture to train. See [Model Architectures](/usage/loading-models) for all options.
```bash theme={null}
--model ViT-B-32
--model ViT-L-14
--model RN50
--model coca_ViT-L-14 # CoCa model
```
Load pretrained weights. Can be a tag (e.g., `laion2b_s34b_b79k`) or a local path.
```bash theme={null}
--pretrained laion2b_s34b_b79k
--pretrained /path/to/checkpoint.pt
```
Load ImageNet pretrained weights for the image encoder (if available).
```bash theme={null}
--pretrained-image
```
### Model Modifications
Override default image input size.
```bash theme={null}
--force-image-size 224
--force-image-size 336 336 # Different height/width
```
Override default text context length.
```bash theme={null}
--force-context-length 77
```
Override patch dropout probability for ViT models. Use 0.5-0.75 for 2-3x speedup.
```bash theme={null}
--force-patch-dropout 0.5 # 50% patch dropout
--force-patch-dropout 0.0 # Disable patch dropout (fine-tuning)
```
Force QuickGELU activation (for compatibility with older checkpoints).
Force separate text tower (CustomTextCLIP architecture).
## Training Hyperparameters
### Batch Size and Epochs
Batch size per GPU. Total batch size = batch\_size × num\_gpus × accum\_freq.
```bash theme={null}
--batch-size 256
```
Number of training epochs.
```bash theme={null}
--epochs 32
```
Gradient accumulation frequency. Simulates larger batch sizes.
```bash theme={null}
--accum-freq 4 # Effective batch = batch_size × 4
```
### Learning Rate
Learning rate. Default depends on model:
* ViT models: 5e-4
* ResNet models: 5e-4
```bash theme={null}
--lr 1e-3
--lr 5e-4
```
Number of warmup steps (linear warmup from 0 to lr).
```bash theme={null}
--warmup 10000
```
Learning rate schedule: `cosine`, `const`, or `const-cooldown`.
```bash theme={null}
--lr-scheduler cosine
--lr-scheduler const # Constant LR after warmup
--lr-scheduler const-cooldown # Constant with cooldown
```
Number of cooldown epochs for `const-cooldown` scheduler.
```bash theme={null}
--lr-scheduler const-cooldown \
--epochs-cooldown 5
```
End learning rate for cooldown.
```bash theme={null}
--lr-cooldown-end 1e-6
```
Power for polynomial cooldown (1.0 = linear).
```bash theme={null}
--lr-cooldown-power 1.0
```
### Optimizer
Optimizer choice. Use `adamw` or `timm/{optimizer}` for timm optimizers.
```bash theme={null}
--opt adamw
--opt timm/sgd
```
Adam beta1 parameter. Default:
* ViT: 0.9
* ResNet: 0.9
```bash theme={null}
--beta1 0.9
```
Adam beta2 parameter. Default:
* ViT: 0.98
* ResNet: 0.999
```bash theme={null}
--beta2 0.98
```
Adam epsilon parameter. Default:
* ViT: 1e-6
* ResNet: 1e-8
```bash theme={null}
--eps 1e-6
```
Weight decay (L2 regularization).
```bash theme={null}
--wd 0.2
--wd 0.1
```
Momentum for timm optimizers (SGD, etc.).
```bash theme={null}
--momentum 0.9
```
### Gradient Clipping
Gradient clipping norm. Prevents gradient explosion.
```bash theme={null}
--grad-clip-norm 1.0
```
## Precision and Memory
### Precision
Training precision: `amp`, `amp_bf16`, `bf16`, `fp16`, `fp32`.
```bash theme={null}
--precision amp # Automatic Mixed Precision (FP16) - Recommended
--precision amp_bf16 # AMP with BFloat16 (A100/H100)
--precision fp32 # Full precision (slow, baseline)
```
### Memory Optimization
Enable gradient checkpointing to reduce memory usage (slower training).
```bash theme={null}
--grad-checkpointing
```
Calculate loss with local features @ global (reduces memory from O(n²) to O(n)).
```bash theme={null}
--local-loss
```
Enable gradient flow through feature gathering (use with --local-loss).
```bash theme={null}
--gather-with-grad
```
Always use `--local-loss` and `--gather-with-grad` together for multi-GPU training (8+ GPUs). See [Distributed Training](/training/distributed-training).
## Data Loading
Number of data loading workers per GPU.
```bash theme={null}
--workers 8 # 8 workers per GPU
```
Recommended: 4-8 workers per GPU for optimal performance.
## Image Preprocessing
Override image normalization mean (RGB).
```bash theme={null}
--image-mean 0.485 0.456 0.406 # ImageNet statistics
```
Override image normalization std (RGB).
```bash theme={null}
--image-std 0.229 0.224 0.225 # ImageNet statistics
```
Image resize interpolation: `bicubic`, `bilinear`, or `random`.
```bash theme={null}
--image-interpolation bicubic
```
Image resize mode: `shortest`, `longest`, or `squash` (inference only).
```bash theme={null}
--image-resize-mode shortest
```
Data augmentation configuration (key-value pairs).
```bash theme={null}
--aug-cfg scale_range=0.08::1.0 ratio_range=0.75::1.33
```
## Model Locking (Transfer Learning)
### Image Tower
Lock (freeze) entire image encoder.
```bash theme={null}
--lock-image
```
Leave last N image tower layer groups unlocked.
```bash theme={null}
--lock-image --lock-image-unlocked-groups 2 # Freeze all but last 2 groups
```
Freeze BatchNorm running statistics in locked layers.
```bash theme={null}
--lock-image-freeze-bn-stats
```
### Text Tower
Lock (freeze) entire text encoder.
```bash theme={null}
--lock-text
```
Leave last N text tower layers unlocked.
```bash theme={null}
--lock-text --lock-text-unlocked-layers 10 # Train last 10 layers
```
Freeze LayerNorm in locked text layers.
```bash theme={null}
--lock-text-freeze-layer-norm
```
## Checkpointing and Logging
### Checkpoints
Save checkpoint every N epochs.
```bash theme={null}
--save-frequency 1 # Save every epoch
--save-frequency 5 # Save every 5 epochs
```
Save most recent checkpoint as `epoch_latest.pt`.
```bash theme={null}
--save-most-recent
```
Delete previous checkpoint after saving new one (saves disk space).
```bash theme={null}
--delete-previous-checkpoint
```
Resume training from checkpoint path or "latest".
```bash theme={null}
--resume /path/to/checkpoint.pt
--resume latest # Resume from latest checkpoint
```
### Logging
Directory for logs and checkpoints.
```bash theme={null}
--logs ./logs/
```
Experiment name (defaults to auto-generated based on timestamp and config).
```bash theme={null}
--name "vit-b32-cc12m-experiment"
```
Logging backends: `tensorboard`, `wandb`, or `tensorboard,wandb`.
```bash theme={null}
--report-to tensorboard
--report-to wandb
--report-to tensorboard,wandb # Both
```
Log training metrics every N steps.
```bash theme={null}
--log-every-n-steps 100
```
### Weights & Biases
W\&B project name.
```bash theme={null}
--wandb-project-name "my-clip-experiments"
```
Notes for W\&B run.
```bash theme={null}
--wandb-notes "Testing new learning rate schedule"
```
## Evaluation
Path to ImageNet validation set for zero-shot evaluation during training.
```bash theme={null}
--imagenet-val /data/imagenet/validation/
```
Path to ImageNet-v2 for additional zero-shot evaluation.
```bash theme={null}
--imagenet-v2 /data/imagenet-v2/
```
Run zero-shot evaluation every N epochs.
```bash theme={null}
--zeroshot-frequency 1 # Every epoch
```
Run validation every N epochs.
```bash theme={null}
--val-frequency 1
```
## CoCa-Specific Parameters
Weight for CoCa contrastive loss.
```bash theme={null}
--coca-contrastive-loss-weight 1.0
```
Weight for CoCa caption generation loss.
```bash theme={null}
--coca-caption-loss-weight 2.0
```
For CoCa fine-tuning on captioning only:
```bash theme={null}
--coca-contrastive-loss-weight 0 \
--coca-caption-loss-weight 1
```
## Distributed Training
URL for distributed training initialization.
```bash theme={null}
--dist-url tcp://localhost:12345
```
Distributed backend: `nccl` (NVIDIA GPU), `hccl` (Ascend NPU), or `gloo` (CPU).
```bash theme={null}
--dist-backend nccl # Default for GPU
```
Use Horovod for distributed training.
```bash theme={null}
--horovod
```
Enable static graph optimization for DDP (PyTorch >= 1.11).
```bash theme={null}
--ddp-static-graph
```
Use synchronized batch normalization across GPUs.
```bash theme={null}
--use-bn-sync
```
## Advanced Options
### Compilation
Compile model with torch.compile() (PyTorch >= 2.0).
```bash theme={null}
--torchcompile
```
TorchScript the model.
```bash theme={null}
--torchscript
```
Trace model with torch.jit.trace (inference only).
```bash theme={null}
--trace
```
### Model Distillation
Teacher model architecture for distillation.
```bash theme={null}
--distill-model ViT-L-14
```
Teacher model pretrained weights.
```bash theme={null}
--distill-pretrained openai
```
### Loss Configuration
Use SigLip (sigmoid) loss instead of standard CLIP loss.
```bash theme={null}
--siglip
```
Distributed loss implementation override.
```bash theme={null}
--loss-dist-impl custom
```
### Remote Syncing
Remote path to sync checkpoints (S3 bucket or filesystem).
```bash theme={null}
--remote-sync s3://my-bucket/checkpoints
```
Sync to remote every N seconds.
```bash theme={null}
--remote-sync-frequency 600 # Sync every 10 minutes
```
Protocol for remote sync: `s3` or `fsspec`.
```bash theme={null}
--remote-sync-protocol s3
```
### Experimental
Use bitsandbytes linear layers for int8 training (experimental).
```bash theme={null}
--use-bnb-linear SwitchBackLinearGlobal
```
### Other
Random seed for reproducibility.
```bash theme={null}
--seed 42
```
Device for training: `cuda` or `cpu`.
```bash theme={null}
--device cuda
```
Override default cache directory for model/tokenizer downloads.
```bash theme={null}
--cache-dir /path/to/cache
```
Enable debug logging.
```bash theme={null}
--debug
```
Log on local master (each node) instead of global master only.
```bash theme={null}
--log-local
```
Copy entire codebase to log directory.
```bash theme={null}
--copy-codebase
```
## Example Configurations
### Small-Scale Training (RN50 on CC3M)
```bash theme={null}
python -m open_clip_train.main \
--train-data "/data/cc3m/train.csv" \
--dataset-type csv \
--csv-img-key filepath \
--csv-caption-key title \
--batch-size 256 \
--precision amp \
--workers 4 \
--warmup 2000 \
--lr 1e-3 \
--wd 0.1 \
--epochs 30 \
--model RN50 \
--save-frequency 5 \
--report-to tensorboard
```
### Medium-Scale Training (ViT-B/32 on CC12M)
```bash theme={null}
torchrun --nproc_per_node 4 -m open_clip_train.main \
--train-data "/data/cc12m/cc12m-{0000..2175}.tar" \
--train-num-samples 10968539 \
--dataset-type webdataset \
--dataset-resampled \
--batch-size 320 \
--precision amp \
--workers 6 \
--imagenet-val /data/imagenet/validation/ \
--warmup 10000 \
--lr 1e-3 \
--wd 0.1 \
--epochs 32 \
--model ViT-B-32 \
--save-frequency 1 \
--zeroshot-frequency 1 \
--local-loss \
--gather-with-grad \
--report-to wandb
```
### Large-Scale Training (ViT-L/14 on LAION-400M)
```bash theme={null}
srun python -u src/open_clip_train/main.py \
--train-data "/data/laion400m/{00000..41455}.tar" \
--train-num-samples 400000000 \
--dataset-type webdataset \
--dataset-resampled \
--batch-size 128 \
--precision amp \
--grad-checkpointing \
--workers 8 \
--warmup 10000 \
--lr 5e-4 \
--wd 0.2 \
--epochs 32 \
--model ViT-L-14 \
--save-frequency 1 \
--zeroshot-frequency 2 \
--local-loss \
--gather-with-grad \
--force-patch-dropout 0.5 \
--report-to wandb \
--remote-sync s3://bucket/checkpoints \
--delete-previous-checkpoint
```
## Recommended Settings by Model
### ViT-B/32
```bash theme={null}
--model ViT-B-32 \
--lr 5e-4 \
--beta1 0.9 \
--beta2 0.98 \
--eps 1e-6 \
--batch-size 256-512 \
--precision amp
```
### ViT-L/14
```bash theme={null}
--model ViT-L-14 \
--lr 5e-4 \
--beta1 0.9 \
--beta2 0.98 \
--eps 1e-6 \
--batch-size 128-256 \
--precision amp \
--grad-checkpointing \
--force-patch-dropout 0.5
```
### RN50
```bash theme={null}
--model RN50 \
--lr 5e-4 \
--beta1 0.9 \
--beta2 0.999 \
--eps 1e-8 \
--batch-size 256-512 \
--precision amp
```
## Next Steps
Apply these configurations to single-node training
Configure distributed training optimizations
Configure data loading and preprocessing
Configure fine-tuning from pretrained models